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The shift towards constructivism in e-learning creates new opportunities in the modern 
school and favors the introducing of novel technologies able to promote a learner-centered ap- 
proach and increase the students’ motivation to learn. Augmented reality (AR) is such a novel 
technology that promotes a new learning experience by integrating real objects, familiar from 
the traditional class, into a computing environment. AR has many valuable features from both 
the teacher’s and learner’s perspective: 3D visualization of complex processes, animation, mul- 
timodal interaction with real objects (visual, auditory and haptic), and augmentation. In turn, 
these features create various learning affordances such as: spatial ability, support to memory 
cognitive processes, experiential learning, collaboration, and motivation (di Serio, Ibanez & 
Kloos, 2012). 

In general, novel technologies are expensive and the costs are not only related to the 
hardware and software platform. The implementation in the traditional school requires addi- 
tional costs for maintenance, content development, and training. Therefore, the first key con- 
cern when evaluating such e-learning systems is to capture the added value brought in by the 
novel technology. This means to go beyond the pedagogical evaluation and to evaluate the 
paradigm against some control condition. For example, it might be possible that a given tech- 
nology works well only for some disciplines but brings no or little added value to others. If we 
go into more detail, the paradigm could work well only for some lessons which focus on the 
understanding of key concepts. In this case, the implementation would be more appropriate for 
a laboratory setting than for a class setting. For example, AR proved to be useful in understand- 
ing the Chemical structure of the atoms, Chemical bonds, and Chemical reactions. The added 
value also varies according to the target user population: a given technology may be appealing 
for young learners in primary school but boring for students in secondary school. 

The second key concern is the usability of a novel technology. The ISO standard 9241- 
210:2010 defined usability as the extent to which a product can be used by specified users to 
achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of 
use. Designing the usability is not an easy task in emerging technologies like AR and VR sys- 
tems, where a lot of resources are spent with 3D visualization features and too little attention is 
paid to the usability of interaction techniques. Usability problems may seriously affect both the 
perceived usefulness and perceived enjoyment. A special attention should be paid to the usabil- 
ity of interaction techniques that imply the manipulation of real objects: if not usable enough 
this might undermine the acceptability of the whole system. 

The third key concern is the acceptability of a new system. The evaluation of technology 
acceptance requires a quantitative approach based on larger models that include several factors, 
such as: characteristics of the system, perceived ease of use, perceived usefulness, relevant 
social aspects, and perceived enjoyment. A problem is the target user: most studies in the e- 
learning domain are only based on students’ perception. This is mainly because quantitative 
methods require a minimum sample size which is easier to collect from students. However, the 
teachers’ opinions are also important as well as the opinion of managers which finally decide if 
it is worth to adopt a novel technology. 
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The learning experience is the user experience with an e-learning system. The ISO stand- 
ard 9241-210:2010 defined the user experience as a person’s perceptions and responses that re- 
sult from the use and/or anticipated use of a product, system, or service. User experience (UX) 
evaluation goes beyond the traditional usability evaluation which is mainly concerned with the 
effectiveness, efficiency and satisfaction of a user that performs specified tasks in a specified 
context of use. UX is also related to the hedonic aspects of using the technology. An enjoyable 
learning experience is increasing the students’ motivation to learn. In the previous work we 
showed that the intention to use an AR teaching platform depends on both its perceived useful- 
ness and perceived enjoyment (Balog & Pribeanu, 2010). 

User experience is a hot field of research that developed rapidly in the last decade. In the 
recent work, Vermeeren et al. (2010) highlighted several challenges for UX research, among 
which we could mention: providing and validating measures for different UX foci and domains, 
providing methods for UX evaluation in collaborative work, need for more effective multi- 
method approaches (which methods work well together). These challenges are quite relevant 
when evaluating how new technologies could positively impact the e-learning systems. 

Providing valid measures for different UX foci in the e-learning domain is not easy. 
There are relatively few studies that analyze the various facets of the hedonic quality and how 
each of them is increasing the motivation to learn. There are also several questions to answer. 
To which extent learning activities should be enjoyable? Which are the features of a novel 
technology that students mostly wish? Which are the attributes that better support the learner’s 
engagement: interesting, captivating, funny? Which are the most suitable measures of enjoy- 
ment in each educational cycle? 

Regarding the methodology for UX evaluation, several issues arise. Traditionally, ap- 
proaches to evaluation are related to either a quantitative or a qualitative paradigm. In this 
respect, mixed methods research is a new paradigm that recently gained considerable attention 
in various application domains such as information systems, social and behavioral sciences. 
Mixed methods approaches combine quantitative and qualitative methods in order to increase 
the confidence in results and get a deeper understanding of specific aspects. Analysis of qualita- 
tive data may reveal many factors that were not taken into consideration in a quantitative model. 
Conversely, qualitative methods could bring valuable insights and enrich the description of 
phenomena captured by a quantitative method. 
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